Image encoding apparatus and image decoding apparatus

ABSTRACT

An input image is subjected to transform by a DCT transforming unit 20. A coefficient analyzing unit 30 compares a predetermined coefficient and coefficient data 120, and obtains low-frequency image data 150 of the input image by a high-frequency coefficient masking unit 50 and an inverse DCT unit 60 on the basis of a result of that comparison. A pixel subsampling unit 70 receives this low-frequency image data 150, and generates a subsampled image on the basis of the aforementioned result of comparison. The subsampled image and coefficient information are transmitted to a decoding side. A decoding apparatus decodes an image on the basis of the subsampled image and the coefficient information.

This is a Division of application Ser. No. 08/956,079 filed Oct. 22.1997. The entire disclosure of the prior application(s) is herebyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image encoding/decoding apparatus,and more particularly to lossy coding with respect to multi-level inputimages.

2. Description of the Related Art

Since images generally comprise very large volumes of data, the imagesare generally compressed by encoding during storage and transmission. Ifthe image data subject to image encoding at that time is largelyclassified into two types, the image data can be classified into, forexample, natural images and artificial images.

The former type is one in which actually existing images have beenconverted into digital data by some means. For instance, an image whichis obtained by reading a photograph by a scanner or by capturing a sceneby a digital camera corresponds to this type. The latter type is one inwhich images which do not actually exist are generated as digital databy some means. For instance, computer graphics and a document which isprepared by a word processor correspond to this type. Hereafter, naturalimages and artificial images are used under these definitions.

Generally, as for natural images, noise tends to be superposed thereonduring digital transform, and their high-frequency components tend todegrade. As a result, the resultant data has a large amount ofinformation in low-order bits, and the number of colors used is alsolarge. In addition, if natural images are subjected to frequencyanalysis, components are liable to concentrate on a low-frequencyregion, and a high-frequency region attenuates.

On the other hand, in the case of artificial images, the amount ofinformation in low-order bits is not large excluding a case in whichnoise is intentionally added thereto, and colors which are used are alsoliable to concentrate on particular colors. In addition, since edges,fine lines, and the like are depicted sharply, a large amount ofimportant information is included in a high-frequency region as well.

Two experimental examples for confirming the above fact are shown inFIGS. 26 to 18. As a first experiment, values in which the square rootsof mean squares of coefficients obtained by discrete cosine transform(DCT) processing were individually determined were examined with respectto a number of images. The results in which the square roots were addedfor the respective eight areas shown in FIG. 26 are shown in the part b)of the same drawing. Since the DCT coefficients are expressed in such amanner that the frequency increases from upper left toward lower right,in FIG. 26 the right-hand side of the x-axis corresponds to a highfrequency. As is apparent from the drawing, in the case of naturalimages, components decrease as the frequency becomes higher, whereas, inthe case of artificial images, components are distributed in spite ofthe frequency.

In a second experiment, adjacent pixel values were fetched from animage, and the statistics of the result of subtraction of a left-handpixel value from a right-hand pixel value was gathered. Hereinafter,this value is referred to as “a previous value differential”. FIG. 28shows the results of the second experiment. As is apparent from thedrawing, in artificial images, the previous value differential isconcentrated in 0 in comparison with natural images. This shows that theprediction accuracy in the prediction of the previous value forpredicting the right-hand pixel value from the left-hand pixel valuebecomes high.

Hereafter, image encoding techniques which are effective for naturalimages and artificial images will be respectively described as first andsecond conventional examples.

First, a description will be given of a conventional encoding techniquewith respect to natural images as a first conventional example. Since anatural image inherently contains a large amount of information, itbecomes necessary to quantize the information by some technique.Therefore, if consideration is given to the efficiency of quantization,since, in the case of a natural image, frequency components areconcentrated in a low-frequency region, quantization in which averageerrors are made small can be realized by quantizing a low-frequencyregion finely and quantizing a high-frequency region coarsely. That is,it is possible to minimize the effect on image quality and reduce theamount of information efficiently.

Frequency transform coding, which is one technique of image encoding,makes use of this characteristic, effects frequency transform of aninput image, and coarsely quantizes information in high-frequency, inparticular. As a typical example of frequency transform coding, it ispossible to cite the DCT method of Joint Photographic Experts Group(JPEG), which is an international standard. Hereafter, a descriptionwill be given of the JPEG-DCT method as a first conventional example.

Before describing the first conventional example, a description will begiven of DCT. The DCT which is used in image encoding is calledtwo-dimensional DCT, to be accurate, and is obtained by independentlyprocessing two one-dimensional DCT in the horizontal direction and thevertical direction. According to “kara seishi gazo no kokusai fugoukahoushiki _JPEG arugorizumu₁₃ (International standard encoding method forcolor still image: JPEG Algorithm)” (Endoh, Interface, 1991. 12, pp.160-182), if an image block subject to transformation is written as x(m,n) and a transformed coefficient block as y(u, v), an 8×8 DCTtransformation formula and an inverse transformation formula for an8-bit image can be written as follows. $\begin{matrix}{\left\lbrack {{Mathematical}\quad {Formula}\quad 1} \right\rbrack \quad} & \quad \\{{y\left( {u,v} \right)} = {\frac{{c(u)}{c(v)}}{4}{\sum\limits_{m = 0}^{7}{\sum\limits_{n = 0}^{7}{\left( {{x\left( {m,n} \right)} - 128} \right)\cos \frac{\left( {{2m} + 1} \right)u\quad \pi}{16}\cos \quad \frac{\left( {{2n} + 1} \right)v\quad \pi}{16}}}}}} & (1) \\{{{x\left( {m,n} \right)} = {{\frac{1}{4}{\sum\limits_{u = 0}^{7}{\sum\limits_{v = 0}^{7}{{c(u)}{c(v)}{y\left( {u,v} \right)}\cos \frac{\left( {{2m} + 1} \right)u\quad \pi}{16}\cos \quad \frac{\left( {{2n} + 1} \right)v\quad \pi}{16}}}}} + 128}}{w\quad h\quad e\quad r\quad e\quad \begin{matrix}{{c(u)},{{c(v)} = {\frac{1}{\sqrt{2}}\left( {u,{v = 0}} \right)}}} \\{= {1\quad a\quad n\quad d\quad o\quad t\quad h\quad e\quad r\quad s}}\end{matrix}}} & (2)\end{matrix}$

FIGS. 29 and 30 are an image lossy encoding apparatus and an image lossydecoding apparatus, respectively, in accordance with the firstconventional example. These drawings are partially taken from FIG. 3 onpage 163 of “kara seishi gazo no kokusai fugouka houshiki _JPEGarugorizumu_(International standard encoding method for color stillimage: JPEG Algorithm)” (ibid.), and terms are modified. In FIGS. 29 and30, reference numeral 10 denotes an image input unit; 20, a DCT unit;35, a coefficient quantizing unit; 45, coefficient output unit; 110,input image data; 120, coefficient data; 170, quantized coefficientdata; 225, a coefficient input unit; 240, an inverse DCT unit; 250, adecoded-image output unit; 260, a coefficient inversely-quantizing unit;320, decoded image data; and 330, inversely-quantized coefficient data.

A description will be given of the various units shown in FIGS. 29 and30. The encoding apparatus in FIG. 29 has the following configuration.The image input unit 10 receives as its input an image from an externalcircuit, and sends the same to the DCT unit 20 as the input image data110. The DCT unit 20 effects DCT processing with respect to the inputimage data 110, and sends the result to the coefficient quantizing unit30 as the coefficient data 120. The coefficient quantizing unit 30effects quantization processing with respect to the coefficient data 120in a predetermined method, and sends the result to a coefficient outputunit 90 as the quantized coefficient data 170. The coefficient outputunit 90 outputs the quantized coefficient data 170 to an externalcircuit.

Next, the decoding apparatus in FIG. 30 has the following configuration.The coefficient input unit 220 receives coefficients as its input, andsends the same to the coefficient inversely-quantizing unit 260 as thequantized coefficient data 170. With respect to the quantizedcoefficient data 170, the coefficient inversely-quantizing unit 260effects inverse quantization, i.e., an inverse transformation of thequantization effected by the coefficient quantizing unit 30, and sendsthe result to the inverse DCT unit 240 as the inversely-quantizedcoefficient data 330. With respect to the inversely-quantizedcoefficient data 330, the inverse DCT unit 240 effects inverse DCTprocessing, i.e., an inverse transformation of the DCT processingeffected by the DCT unit 20, and sends the result to the decoded-imageoutput unit 250 as the decoded image data 320. The decoded-image outputunit 250 outputs the decoded image data 320 to an external circuit.

The above-described configuration is a part of the first conventionalexample, and a general configuration is arranged such that the quantizedcoefficient data 170 is generally subjected to variable-length coding,such as Huffman coding and QM coding, by the encoding apparatus, while adecoding corresponding to the variable-length coding is effected by thedecoding apparatus, thereby obtaining the quantized coefficient data170. Since these portions are irrelevant to the essence of the presentinvention, and the omission of these portions does not impair theessence of the first conventional example, a description thereof will beomitted here.

On the basis of the above-described configuration, a description will begiven of the operation of the first conventional example. FIGS. 31 and32 are flowcharts illustrating the operation of the conventionalexample.

First, referring to FIG. 31, a description will be given of the encodingprocedure in accordance with the first conventional example. In S10, animage is inputted to the image input unit 10 from an external circuit,and the input image data 110 is obtained. In S20, the DCT unit 20effects DCT processing to obtain the coefficient data 120. In S35, thecoefficient quantizing unit 30 effects quantization processing withrespect to the coefficient data 120 in a predetermined method, therebyobtaining the quantized coefficient data 170. In S75, the coefficientoutput unit 90 outputs the quantized coefficient data 170 to an externalcircuit. In S80, a determination is made as to whether or not all theprocessing of the input image data 110 inputted has been completed, andif not completed, the operation returns to S10, and if completed, theencoding procedure ends.

Next, referring to FIG. 32, a description will be given of the decodingprocedure in accordance with the first conventional example. In S115,coefficients are inputted to the coefficient input unit 220 from anexternal circuit, and the quantized coefficient data 170 is obtained. InS125, the coefficient inversely-quantizing unit 260 effects inversequantization processing to obtain the inversely-quantized coefficientdata 330. In S130, the inverse DCT unit 240 effects inverse DCTprocessing with respect to the inversely-quantized coefficient data,thereby obtaining the decoded image data 320. In S140, the decoded-imageoutput unit 250 outputs the decoded image data 320 to an externalcircuit. In S150, a determination is made as to whether or not all theprocessing of the quantized coefficient data 170 inputted has beencompleted, and if not completed, the operation returns to S115, and ifcompleted, the decoding procedure ends.

A description will be given of the quantization processing which iseffected by the coefficient quantizing unit 35 in the above-describedoperation. As described above, in a general frequency transform codingscheme, high-frequency components are coarsely quantized as comparedwith low-frequency components. In the JPEG-DCT method, a linearquantization of the formula shown below is used. Here, round is afunction which returns an integer which is closest to an argument.

[Mathematical Formula 2]

(Quantization coefficient)=round ((DCT coefficient)/(quantizationstep)  (3)

FIGS. 33A and 33B show a recommended quantization table of the JPEG-DCTmethod (source: ibid., FIG. 9 on page 167 of “kara seishi gazo nokokusai fugouka houshiki _JPEG arugorizumu_(International standardencoding method for color still image: JPEG Algorithm)”). In thedrawing, numerals represent quantization steps, and the greater thenumeral value, the more coarsely quantization is effected. In the sameway as the DCT coefficients in Formula (1), the quantization table iswritten in such a manner that the frequency becomes higher from upperleft toward lower right; hence, it follows that high-frequencycomponents, in particular, are quantized coarsely.

Next, a conventional encoding technique concerning artificial imageswill be described as a second conventional example. In artificialimages, since the same colors often locally appear in a space as shownin FIG. 28, predictive coding in which prediction of pixel values on thebasis of neighboring pixels and encoding of prediction errors arecombined is effective. Hereafter, as a typical example of predictivecoding, the spatial method, which is a lossless coding method set forthin the aforementioned international JPEG, will be described as thesecond conventional example.

Before specifically describing the second conventional example, adescription will be given of predictive coding. Predictive coding is atechnique in which a pixel value of a pixel subjected to coding next isestimated, and a prediction error obtained by the following formula isencoded.

[Mathematical Formula 3]

 (Prediction error)=(actual pixel value)−(predicted value)  (4)

In artificial images, since prediction errors are concentrated in 0 asshown in FIG. 27, the amount of codes can be reduced as compared withnatural images. In addition, particularly in lossless predictive coding,amount-of-coding control cannot be effected; however, there is nopossibility of the degradation of image quality.

Hereafter, a specific description will be given of the secondconventional example. FIGS. 34 and 35 are an image lossy encodingapparatus and an image lossy decoding apparatus, respectively, inaccordance with the second conventional example. These drawings arepartially taken from ibid., FIG. 17 on page 173 of “kara seishi gazo nokokusai fugouka houshiki _JPEG arugorizumu_(International standardencoding method for color still image: JPEG Algorithm)”, a decodingapparatus is added, and terms are modified. In the drawings, thoseportions which are similar to those of FIGS. 29 and 30 will be denotedby the same reference numerals, and a description thereof will beomitted. Reference numeral 25 denotes a predicting unit; 46, aprediction-error output unit; 226, a prediction-error input unit; and171, prediction error data.

A description will be given of the various units shown in FIGS. 34 and35. The encoding apparatus in FIG. 34 has the following configuration.The predicting unit 25 predicts a pixel value to be encoded next byusing the input image data 110, and sends a difference with an actualpixel value to the prediction-error output unit 46 as the predictionerror data 171.

The decoding apparatus in FIG. 35 has the following configuration. Theprediction-error input unit 226 receives prediction errors as its input,and sends the same to the predicting unit 25 as the prediction errordata 171. Although the predicting unit 25 is identical to the predictingunit 25 of the encoding apparatus, but differs from the same in thatreference is made to a decoded image for predicting a next pixel.

On the basis of the above-described configuration, a description will begiven of the operation in accordance with the second conventionalexample. FIGS. 36 and 37 are flowcharts illustrating the operation ofthe conventional example.

First, referring to FIG. 36, a description will be given of the encodingprocedure in accordance with the first conventional example. Thoseportions which are similar to those of FIG. 31 are denoted by the samereference numerals, and a description thereof will be omitted. In S25,the predicting unit 25 computes a prediction error in accordance withFormula (4). In S76, the prediction-error output unit 46 outputs to anexternal circuit the prediction error data 171 computed in S25.

Next, referring to FIG. 37, a description will be given of the decodingprocedure in accordance with the first conventional example. Thoseportions which are similar to those of FIG. 32 are denoted by the samereference numerals, and a description thereof will be omitted. In S116,the prediction-error input unit 226 receives as its input the predictionerror from an external circuit. In S135, the predicting unit 25 computesa pixel value by the addition of the predicted value and the predictionerror.

A description will be given of the prediction error computationprocessing in the description of the operation. In the JPEG-Spatialmethod, it is decided that one of the seven predictors shown in FIG. 38be used. For example, in a case where a is selected as a predictionformula, it suffices if the value of a pixel which is a left-sideneighbor to a pixel x to be encoded from now on is set as a predictedvalue.

Although the first and second conventional examples have been describedabove, it is shown below that it is difficult to effect encodingefficiently by either one of the first and second conventional examplesirrespective of the distinction between natural images and artificialimages.

Since, in an artificial image, important information is included inhigh-frequency components as well, if quantization is effected as shownin FIGS. 33A and 33B in which a high-frequency region is coarselyquantized, the degradation of the image quality, e.g., mosquito noise,occurs. An example of mosquito noise which occurred due to thequantization table shown in FIG. 33A is shown in FIGS. 39A and 39B. FIG.39A shows an input image, while FIG. 39B shows a decoded image. In afrequency transform coding scheme such as JPEG-DCT, such noise makes itdifficult to reduce the amount of codes while maintaining the imagequality with respect to an artificial image. This state is shown in FIG.40.

On the other hand, in the case of a natural image, pixel values differeven among neighboring pixels due to the effect of noise, so that theamount of codes does not diminish in the lossless predictive coding suchas JPEG-Spatial method. This state is shown in FIG. 41. In addition, inthe lossless coding, the image quality and the amount of codes cannot betraded off, so that amount-of-coding control cannot be effected. Sincethis drawback directly affects the capacity of a storage medium, acommunication band, and the like, the structuring of the system is madedifficult.

Thus, in the first and second conventional examples, images which cannotbe encoded effectively are present. To overcome this problem, atechnique in which lossy encoding and lossless encoding are selectivelyused for respective portions is conceivable. As such an example,Japanese Patent Application Laid-Open No. 113145/1994 is known.Hereafter, the invention disclosed in that publication will be describedas a third conventional example.

FIG. 42 is a schematic diagram of an image processing apparatus inaccordance with the third conventional example. In this drawing, aportion of FIG. 1 in that publication is omitted in such a way as not toimpair the purport of Japanese Patent Application Laid-Open No.113145/1994, and terms are modified. In the drawing, reference numeral15 denotes an artificial-image input unit; 16, a natural-image inputunit; 90, an artificial-image encoding unit; 91, a natural-imageencoding unit; 92, an artificial-image storage unit; 93, a natural-imagestorage unit; 94, an artificial-image decoding unit; 95, a natural-imagedecoding unit; 96, an image composing unit; 112, input artificial-imagedata; 113, input natural-image data; 114, encoded artificial-image data;115, encoded natural-image data; 116, decoded artificial-image data; and117, decoded natural-image data.

A description will be given of the various units shown in FIG. 42, theartificial-image input unit 15 and the natural-image input unit 16respectively receive as their inputs an artificial image and a naturalimage from external circuits, and send them to the artificial-imageencoding unit 90 and the natural-image encoding unit 91 as the inputartificial-image data and the input natural-image data 113,respectively. The artificial-image encoding unit 90 and thenatural-image encoding unit 91 effect encoding with respect to the inputartificial-image data and the input natural-image data 113 bypredetermined techniques, respectively, and send the results to theartificial-image storage unit 92 and the natural-image storage unit 93as the encoded artificial-image data 114 and the encoded natural-imagedata 115, respectively. The artificial-image storage unit 92 and thenatural-image storage unit 93 temporarily store the encodedartificial-image data 114 and the encoded natural-image data 115,respectively, and send them to the artificial-image decoding unit 94 andthe natural-image decoding unit 95, respectively. With respect to theencoded artificial-image data 114 and the encoded natural-image data115, the artificial-image decoding unit 94 and the natural-imagedecoding unit 95 respectively effect decoding processings correspondingto the encoding effected by the artificial-image encoding unit 90 andthe natural-image encoding unit 91, and send the results to the imagecomposing unit 96 as the decoded artificial-image data 116 and thedecoded natural-image data 117, respectively. The image composing unit96 combines the decoded artificial-image data 116 and the decodednatural-image data 117.

In the above description, it is stated in the first embodiment of theaforementioned patent that the encoding which is effected by theartificial-image encoding unit 90 “has the function of a lossless methodsuch as the run-length coding method.” In addition, it is also stated inthe first embodiment of that patent that the encoding which is effectedby the natural-image encoding unit 91 is “an image compression systemsuch as JPEG” It should be noted that JPEG referred to in that patentrefers to the JPEG-DCT method referred to in this description.

It has already been pointed out that since the first and secondconventional examples are designed specifically for natural images andartificial images, respectively, it is difficult to handle both types ofimages efficiently by either one of the independent techniques.

In the third conventional example, since a natural image and anartificial image are encoded and decoded in parallel by totallydifferent methods, the processing times in both processings generally donot coincide. For this reason, it is impossible to produce an output toan external circuit until all the encoded data are gathered duringencoding, and until all the image data are gathered during decoding.Hence, at least one image portion of a code buffer is required for theencoding apparatus, while at least one image portion of an image bufferis similarly required for the decoding apparatus. These are unnecessaryconfigurations in the case of an image encoding/decoding apparatushaving a method of only one system.

In addition, since both the encoding apparatus and the decodingapparatus have two systems or more, an increase in the scale of theapparatus results. Further, since the image is expressed by a pluralityof totally different codes, the handling of codes becomes complex duringsuch as transmission or storage. Still further, with respect to theimage quality of a decoded image as well, noise can possibly occur in aportion where the encoding method is changed over.

SUMMARY OF THE INVENTION

The present invention has been devised in view of the above-describedcircumstances, and its object is to provide a single encoding apparatusand a decoding apparatus capable of effective compression irrespectiveof the distinction between a natural image and an artificial image.

To attain the above object, the present invention adopts the followingconfigurations. First, a description will be given of the invention ofthe image encoding apparatus.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;frequency transforming means for effecting frequency transform fordetermining frequency components of the image inputted by said imageinput means; threshold processing means for effecting thresholdprocessing of the frequency components determined by said frequencytransforming means; low-frequency image output means for outputting animage of low-frequency components of the image inputted by said imageinput means, in correspondence with a result of threshold processing bysaid threshold processing means; pixel subsampling means for effectingpredetermined subsampling processing with respect to the image outputtedby said low-frequency image output means, in correspondence with theresult of threshold processing by said threshold processing means;coefficient-information output means for outputting the result ofthreshold processing by said threshold processing means; andsubsampled-image output means for outputting the image subjected tosubsampling processing by said pixel subsampling means.

In this configuration, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes. To determine the optimum resolution, frequencyanalysis is performed, and the subsampling processing of pixels iseffected on the basis of the result of this analysis.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;frequency transforming means for effecting frequency transform fordetermining frequency components of the image inputted by said imageinput means; threshold processing means for effecting thresholdprocessing of the frequency components determined by said frequencytransforming means; high-frequency coefficient masking means forreplacing high-frequency components with 0s of the frequency componentsdetermined by said frequency transforming means, in correspondence witha result of threshold processing by said threshold processing means;inversely transforming means for effecting inverse frequency transformin which the frequency components with the high-frequency componentsreplaced into 0s by said high-frequency coefficient masking means areconverted into an image; pixel subsampling means for effectingpredetermined subsampling processing with respect to the image convertedby said inversely transforming means, in correspondence with the resultof threshold processing by said threshold processing means;coefficient-information output means for outputting the result ofthreshold processing by said threshold processing means; andsubsampled-image output means for outputting the image subjected tosubsampling processing by said pixel subsampling means.

In this configuration as well, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;frequency transforming means for effecting frequency transform fordetermining frequency components of the image inputted by said imageinput means; threshold processing means for-effecting thresholdprocessing of the frequency components determined by said frequencytransforming means; pixel subsampling means for effecting predeterminedsubsampling processing with respect to the image inputted by said imageinput means, in correspondence with the result of threshold processingby said threshold processing means; coefficient-information output meansfor outputting the result of threshold processing by said thresholdprocessing means; and subsampled-image output means for outputting theimage subjected to subsampling processing by said pixel subsamplingmeans.

In this configuration as well, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;pseudo-decoded-image generating means for generating a pseudo-decodedimage by subjecting the image inputted by said image input means topredetermined subsampling processing and predetermined interpolationprocessing; coefficient analyzing means for determining a subsamplingrate on the basis of an error between the pseudo-decoded image generatedby said pseudo-decoded-image generating means and the image inputted bysaid image input means; pixel subsampling means for effectingpredetermined subsampling processing with respect to the image inputtedby said image input means, in correspondence with the subsampling ratedetermined by said coefficient analyzing means; coefficient-informationoutput means for outputting the subsampling rate determined by saidcoefficient analyzing means; and subsampled-image output means foroutputting the image subjected to subsampling processing by said pixelsubsampling means.

In this configuration as well, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

Further, in accordance with the invention, in the image encodingapparatus, the error used in said coefficient analyzing means is amaximum value of a pixel value error, an absolute value of the error,and a squared value of the error, or one of a dynamic range, a variance,and an SN ratio.

Further, in accordance with the invention, in the image encodingapparatus, the predetermined interpolation processing by saidpseudo-decoded-image generating means is one of nearest-neighborinterpolation, 4-point linear interpolation, 9-point second-orderinterpolation, cubic convolution interpolation, and low-pass filterprocessing.

In accordance with the invention, there is provided an image encodingapparatus comprising: code input means for inputting codes obtained bysubjecting an image to frequency transform and entropy coding; entropydecoding means for obtaining frequency components by subjecting thecodes inputted by said code input means to decoding which is an inversetransformation of entropy coding effected with respect to the codes;threshold processing means for effecting threshold processing withrespect to the frequency components obtained by said entropy decodingmeans; high-frequency coefficient masking means for replacinghigh-frequency components with 0s of the frequency components obtainedby said entropy decoding means, in correspondence with a result ofthreshold processing by said threshold processing means; inverselytransforming means for effecting inverse frequency transform in whichthe frequency components with the high-frequency components replacedinto 0s by said high-frequency coefficient masking means are convertedinto an image; pixel subsampling means for effecting predeterminedsubsampling processing with respect to the image converted by saidinversely transforming means, in correspondence with the result ofthreshold processing by said threshold processing means;coefficient-information output means for outputting the result ofthreshold processing by said threshold processing means; andsubsampled-image output means for outputting the image subjected tosubsampling processing by said pixel subsampling means.

Further, in accordance with the invention, in the image encodingapparatus, the decoding by said entropy decoding means is one of Huffmancoding, arithmetic coding, and QM coding.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;frequency transforming means for effecting frequency transform fordetermining frequency components of the image inputted by said imageinput means; threshold processing means for effecting thresholdprocessing of the frequency components determined by said frequencytransforming means; high-frequency coefficient masking means forreplacing high-frequency components with 0s of the frequency componentsobtained by said frequency transforming means, in correspondence with aresult of threshold processing by said threshold processing means;inversely transforming means for effecting inverse frequency transformin which the frequency components with the high-frequency componentsreplaced into 0s by said high-frequency coefficient masking means areconverted into an image; pixel subsampling means for effectingpredetermined subsampling processing with respect to the image convertedby said inversely transforming means, in correspondence with the resultof threshold processing by said threshold processing means; datacomposing means for combining the subsampled image obtained by saidpixel subsampling means and the result of threshold processing by saidthreshold processing means; and composite-data output means foroutputting composite data composed by said data composing means.

In this configuration as well, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

In accordance with the invention, there is provided an image encodingapparatus comprising: image input means for inputting an image;coefficient-information input means for inputting coefficientinformation; frequency transforming means for effecting frequencytransform for determining frequency components of the image inputted bysaid image input means; high-frequency coefficient masking means forreplacing high-frequency components with 0s of the frequency componentsdetermined by said frequency transforming means, in correspondence withthe coefficient information inputted by said coefficient-informationinput means; inversely transforming means for effecting inversefrequency transform in which the frequency components with thehigh-frequency components replaced into 0s by said high-frequencycoefficient masking means are converted into an image; pixel subsamplingmeans for effecting predetermined subsampling processing with respect tothe image converted by said inversely transforming means, incorrespondence with the coefficient information inputted by saidcoefficient-information input means; coefficient-information outputmeans for outputting the coefficient information inputted by saidcoefficient-information input means; and subsampled-image output meansfor outputting the image subjected to subsampling processing by saidpixel subsampling means.

In this configuration as well, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

Further, in accordance with the invention, the image encoding apparatusfurther comprises: image encoding means for effecting image encodingwith respect to the subsampled image outputted by said subsampled-imageoutput means.

Further, in accordance with the invention, in the image encodingapparatus, the image encoding effected by said image encoding means isone of or both of lossless coding and predicting coding.

Further, in accordance with the invention, the image encoding apparatusfurther comprises: coefficient-image encoding means for effectingentropy coding with respect to the coefficient image outputted by saidcoefficient-image output means.

Further, in accordance with the invention, in the image encodingapparatus, the frequency transform effected by said frequencytransforming means and said inversely transforming means is one ofdiscrete cosine transform, Fourier transform, discrete sine transform,subband transform, and wavelet transform.

Further, in accordance with the invention, in the image encodingapparatus, the threshold processing by said threshold processing meansis threshold processing in which a predetermined quantization table isset as the threshold.

Further, in accordance with the invention, in the image encodingapparatus, the quantization table used by said threshold processingmeans can be set by an external circuit.

Further, in accordance with the invention, in the image encodingapparatus, said high-frequency coefficient masking means replaces acomponent greater than a maximum frequency component with a 0 by meansof said threshold processing means.

Further, in accordance with the invention, in the image encodingapparatus, the subsampling processing by said pixel subsampling means iseffected in proportion to a ratio which is derived from a distributionof maximum frequencies within a block or frequency components which arenot 0s.

Further, in accordance with the invention, in the image encodingapparatus, a ratio of subsampling processing effected by said pixelsubsampling means is quantization to a predetermined value set inadvance.

Further, in accordance with the invention, in the image encodingapparatus, the predetermined subsampling processing by said pixelsubsampling means is one of leaving pixels in lattice form, effectingthe subsampling processing at identical rates for a vertical directionand a horizontal direction, effecting the subsampling processing suchthat pixels which remain become substantially equidistanced, andpreferentially leaving peak values in neighboring pixels.

Further, in accordance with the invention, in the image encodingapparatus, the subsampling processing by said pixel subsampling means isthe thinning out of the same pixels which were previously thinned out ina case where the image inputted by said image input means were alreadysubjected to encoding by said image encoding apparatus.

Further, in accordance with the invention, the image encoding apparatusfurther comprises: pixel-value quantizing means for quantizing a pixelvalue of the image subjected to subsampling processing by said pixelsubsampling means.

Further, in accordance with the invention, in the image encodingapparatus, said pixel-value quantizing means changes a quantization stepin correspondence with a result of threshold processing by saidthreshold processing means, or changes the quantization step incorrespondence with a magnitude of the threshold used by said thresholdprocessing means.

Further, in accordance with the invention, the image encoding apparatusfurther comprises: image determining means for determining the thresholdused by said threshold processing means by performing predeterminedanalysis with respect to the image inputted by said image input means.

Further, in accordance with the invention, in the image encodingapparatus, said image determining means determines a difference betweena natural image and an artificial image, and in the case of theartificial image sets the threshold to a 0 and effects control so as toprevent the occurrence of a frequency component which is set to a 0 inthe threshold processing by said threshold processing means.

Further, in accordance with the invention, in the image encodingapparatus, the predetermined analysis processing by said imagedetermining means involves measurement of a dynamic range of the pixelvalues, measurement of a histogram of the pixel values, measurement ofan entropy of lower bits of the pixel values, measurement of thesharpness of an edge, measurement of the size of a line, measurement ofa frequency component, designation from an external circuit, anddetection of at least one component from among an edge, a pattern, agradation, and a line.

Next, a description will be given of an image decoding apparatus.

In accordance with the invention, there is provided an image decodingapparatus comprising: coefficient-information input means for inputtingcoefficient information; subsampled-image input means for inputting asubsampled image; coefficient interpolating means for computing afrequency component by a predetermined technique in correspondence withthe subsampled image inputted by said subsampled-image input means andthe coefficient information inputted by said coefficient-informationinput means; inversely transforming means for effecting inversefrequency transform so as to convert the frequency component computed bysaid coefficient interpolating means into an image; and decoded-imageoutput means for outputting the image converted by said inverselytransforming means.

In this configuration, it is possible to decode image data which hasbeen compressed by effecting adaptive subsampling in correspondence withfrequency analysis.

In accordance with the invention, there is provided an image decodingapparatus comprising: coefficient-information input means for inputtingcoefficient information for each block which is a fixed region of animage; subsampled-image input means for inputting a subsampled image foreach block; pixel-value interpolating means for interpolating a pixelvalue by a predetermined technique in correspondence with the subsampledimage inputted by said subsampled-image input means and the coefficientinformation inputted by said coefficient-information input means; anddecoded-image output means for outputting the image interpolated by saidpixel-value interpolating means.

In this configuration as well, it is possible to decode image data whichhas been compressed by effecting adaptive subsampling in correspondencewith frequency analysis.

Further, in accordance with the invention, in the image decodingapparatus, the predetermined technique used by said pixel-valueinterpolating means is one of nearest-neighbor interpolation, 4-pointlinear interpolation, 9-point second-order interpolation, cubicconvolution interpolation, and low-pass filter processing.

In accordance with the invention, there is provided an image decodingapparatus comprising: composite-data input means for inputting compositedata which is data combining coefficient information and a subsampledimage; data decomposing means for decoding the composite data inputtedby said composite-data input means into the subsampled image and thecoefficient information; coefficient interpolating means for computing afrequency component by a predetermined technique, in correspondence withthe subsampled image and the coefficient information which weredecomposed by said data decomposing means; inversely transforming meansfor effecting inverse frequency transform in which the frequencycomponent computed by said coefficient interpolating means is convertedinto the image; and decoded-image output means for outputting the imageconverted by said inversely transforming means.

In this configuration as well, it is possible to decode image data whichhas been compressed by effecting adaptive subsampling in correspondencewith frequency analysis.

Further, in accordance with the invention, the image decoding apparatusfurther comprises: image decoding means for decoding into an image acode subjected to image encoding with respect to the subsampled image,wherein said subsampled-image input means inputs as the subsampled imagethe image decoded by said image decoding means.

Further, in accordance with the invention, in the image decodingapparatus, the decoding effected by said image decoding means is inverseprocessing of lossless coding or inverse processing of predictivecoding.

Further, in accordance with the invention, the image decoding apparatusfurther comprises: pixel-value correcting means for replacing a pixel,which is included in the subsampled image inputted by saidsubsampled-image input means of the image converted by said inverselytransforming means, with the pixel value of the subsampled image,wherein said decoded-image output means outputs the image corrected bysaid pixel-value correcting means.

Further, in accordance with the invention, in the image decodingapparatus, the frequency transform effected by said inverselytransforming means and said inversely transforming means is one ofdiscrete cosine transform, Fourier transform, discrete sine transform,subband transform, and wavelet transform.

Further, in accordance with the invention, in the image decodingapparatus, the coefficient interpolation effected by said coefficientinterpolating means is one of the solving of a simultaneous system oflinear equations concerning frequency coefficients and pixel values,computation of an inverse matrix determined in advance with respect tothe simultaneous system of linear equations concerning frequencycoefficients and pixel values, and low-pass filtering of the subsampledimage or approximate processing.

In accordance with the invention, there is provided an imageencoding/decoding apparatus comprising: image input means for inputtingan image; frequency transforming means for effecting frequency transformfor determining frequency components of the image inputted by said imageinput means; threshold processing means for effecting thresholdprocessing of the frequency components determined by said frequencytransforming means; high-frequency coefficient masking means forreplacing high-frequency components with 0s of the frequency componentsdetermined by said frequency transforming means, in correspondence witha result of threshold processing by said threshold processing means;first inversely transforming means for effecting inverse frequencytransform in which the frequency components with the high-frequencycomponents replaced into 0s by said high-frequency coefficient maskingmeans are converted into an image; pixel subsampling means for effectingpredetermined subsampling processing with respect to the image convertedby said first inversely transforming means, in correspondence with theresult of threshold processing by said threshold processing means;coefficient-information output means for outputting the result ofthreshold processing by said threshold processing means;subsampled-image output means for outputting the image subjected tosubsampling processing by said pixel subsampling means;coefficient-information input means for inputting coefficientinformation which is a result of threshold processing outputted by saidcoefficient-information output means; subsampled-image input means forinputting the subsampled image outputted by said subsampled-image outputmeans; coefficient interpolating means for computing a frequencycomponent by a predetermined technique in correspondence with thesubsampled image inputted by said subsampled-image input means and thecoefficient information inputted by said coefficient-information inputmeans; second inversely transforming means for effecting inversefrequency transform so as to convert the frequency component computed bysaid coefficient interpolating means into an image; and decoded-imageoutput means for outputting the image converted by said second inverselytransforming means.

In this configuration, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes. To obtain the optimum resolution, frequencyanalysis is performed, and the subsampling processing of pixels iseffected on the basis of the result of this analysis. In addition, it ispossible to decode image data which has been compressed by effectingadaptive subsampling in correspondence with the frequency analysis.

In accordance with the invention, there is provided an image encodingmethod comprising: step 1 for inputting an image; step 2 for effectingfrequency transform for determining frequency components of the imageinputted in said step 1; step 3 for effecting threshold processing ofthe frequency components determined in said step 2; step 4 for replacinghigh-frequency components with 0s of the frequency components determinedin said step 2, in correspondence with a result of threshold processingin said step 3; step 5 for effecting inverse frequency transform inwhich the frequency components with the high-frequency componentsreplaced into 0s in said step 4 are converted into an image; step 6 foreffecting predetermined subsampling processing with respect to the imageconverted in said step 5, in correspondence with the result of thresholdprocessing in said step 3; step 7 for outputting the result of thresholdprocessing in said step 3; and step 8 for outputting the image subjectedto subsampling processing in said step 6.

In this configuration, by representing an image with an optimumresolution, it is possible to suppress redundant components and reducethe amount of codes.

In accordance with the invention, there is provided an image decodingmethod comprising: step 1 for inputting coefficient information; step 2for inputting a subsampled image; step 3 for computing a frequencycomponent by a predetermined technique in correspondence with thesubsampled image inputted in said step 2 and the coefficient informationinputted in said step 1; step 4 for effecting inverse frequencytransform so as to convert the frequency component computed said step 3;and step 5 for outputting the image converted step 4.

In this configuration as well, it is possible to decode image data whichhas been compressed by effecting adaptive subsampling in correspondencewith the frequency analysis.

The above and other objects and features of the present invention willbe more apparent from the following description taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an image encoding apparatusin accordance with a first embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating an image decoding apparatusin accordance with the first embodiment of the present invention;

FIG. 3 is a flowchart illustrating an example of the operation ofencoding processing by the image encoding apparatus in accordance withthe first embodiment of the present invention;

FIG. 4 is a flowchart illustrating an example of the operation ofdecoding processing by the image decoding apparatus in accordance withthe first embodiment of the present invention;

FIGS. 5A to 5D are diagrams explaining subsampling processing in thefirst embodiment of the present invention;

FIGS. 6A to 6E are diagrams explaining coefficient data processing inthe first embodiment of the present invention;

FIG. 7 is an explanatory diagram concerning errors of pixel values in adecoded image in accordance with a JPEG-DCT method;

FIG. 8 is a diagram explaining information for subsampling for encoding;

FIG. 9 is a diagram explaining information for subsampling for encoding;

FIG. 10 is a schematic diagram illustrating an example of extension inthe first embodiment of the present invention;

FIG. 11 is a schematic diagram illustrating an example of extension inthe first embodiment of the present invention;

FIG. 12 is a schematic diagram illustrating another example of extensionin the first embodiment of the present invention;

FIG. 13 is a schematic diagram illustrating an example of simplificationin the first embodiment of the present invention;

FIG. 14 is a schematic diagram illustrating another example ofsimplification in the first embodiment of the present invention;

FIG. 15 is a schematic diagram illustrating still another example ofsimplification in the first embodiment of the present invention;

FIG. 16 is a schematic diagram illustrating a further example ofsimplification in the first embodiment of the present invention;

FIG. 17 is a diagram illustrating an example of simplification in thefirst embodiment of the present invention;

FIG. 18 is a diagram illustrating another example of simplification inthe first embodiment of the present invention;

FIG. 19 is a schematic diagram illustrating a further example ofsimplification in the first embodiment of the present invention;

FIG. 20 is an explanatory diagram illustrating an example of the resultsof an experiment in accordance with the first embodiment of the presentinvention;

FIG. 21 is a schematic diagram illustrating an image encoding apparatusin accordance with a second embodiment of the present invention;

FIG. 22 is a schematic diagram illustrating an image decoding apparatusin accordance with the second embodiment of the present invention;

FIG. 23 is an explanatory diagram schematically illustrating thecomparison of the amount of codes between the first embodiment and afirst conventional example;

FIGS. 24A to 24C are explanatory diagrams illustrating the comparison ofthe degradation of image quality between the first embodiment and thefirst conventional example;

FIGS. 25A and 25B are explanatory diagrams illustrating the comparisonof the degradation of image quality between the first embodiment and thefirst conventional example;

FIG. 26 is an explanatory diagram of an example of an experimentillustrating the characteristics of an image;

FIG. 27 is an explanatory diagram of an example of an experimentillustrating the characteristics of images;

FIG. 28 is an explanatory diagram of an example of an experimentillustrating the characteristics of images;

FIG. 29 is a schematic diagram illustrating an image encoding apparatusin accordance with the first conventional example of the presentinvention;

FIG. 30 is a schematic diagram illustrating an image decoding apparatusin accordance with the first conventional example of the presentinvention;

FIG. 31 is a flowchart illustrating an example of the operation ofencoding processing by the image encoding apparatus in accordance withthe first conventional example;

FIG. 32 is a flowchart illustrating an example of the operation ofdecoding processing by the image decoding apparatus in accordance withthe first conventional example;

FIGS. 33A and 33B are explanatory diagrams of an example of aquantization table used in the first conventional example;

FIG. 34 is a schematic diagram illustrating an image encoding apparatusin accordance with a second conventional example of the presentinvention;

FIG. 35 is a schematic diagram illustrating an image decoding apparatusin accordance with the second conventional example of the presentinvention;

FIG. 36 is a flowchart illustrating an example of the operation ofencoding processing by the image encoding apparatus in accordance withthe second conventional example;

FIG. 37 is a flowchart illustrating an example of the operation ofdecoding processing by the image decoding apparatus in accordance withthe second conventional example;

FIG. 38 is an explanatory diagram of predictors used in the secondconventional example;

FIGS. 39A and 39B are explanatory diagrams on mosquito noise;

FIG. 40 is an explanatory diagram of an example of an experiment inaccordance with the first conventional example;

FIG. 41 is an explanatory diagram of an example of an experiment inaccordance with the second conventional example; and

FIG. 42 is a schematic diagram illustrating a third conventionalexample.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereafter, a description will be given of the embodiments of the presentinvention. First, a first embodiment using DCT will be described, and asecond embodiment using a frequency transform technique other than DCTwill then be described.

(First Embodiment)

Prior to giving a specific description of the first embodiment of thepresent invention, a description will be given of a basic concept of thepresent invention. The amount of data of a digital image is determinedby the resolution and the number of bits per pixel. As for an imageformat, both the resolution and the number of bits are generally fixedby constants.

However, the amount of information of an image changes locally. Forinstance, where there is no change in the pixel value, a maximumresolution is not required, and the number of bits can be restricted.Namely, this means that a fixed image format contains redundantinformation.

In the case of a natural image, in particular, a maximum resolution ofthe image is restricted by the frequency characteristic and theresolution of a device for effecting digital transform. For example, ina case where a digital image inputted by a scanner having a resolution sis managed by an image format having a resolution 2s, the amount ofessentially significant pixels is merely s²/(2s)²=¼. This phenomenonbecomes noticeable in obtaining a high resolution for an output device,enlargement processing of an image, and the like.

Accordingly, a case is considered in which an image is represented withan optimum resolution. A resolution required for a digital image isdependent upon a maximum frequency which the image possesses. Forexample, a pitch p of the resolution cannot be made shorter than a halflength T/2 of a period T of the maximum frequency which the imagepossesses. If this is considered conversely, it can be said that animage which does not use up to a maximum frequency allowed by theresolution contains redundant pixels. Even if such redundant pixels arethinned out, the pixels can be interpolated afterwards by neighboringpixels insofar as the maximum frequency is known.

In the present invention, redundant components are suppressed and theamount of codes is reduced by representing an image with an optimumresolution on the basis of the above-described basic principle. Thetransform of the resolution into the aforementioned optimum resolutionis realized by the subsampling of pixels. In addition, analyticalprocessing for obtaining the optimum resolution is effected by frequencyanalysis. Encoding and decoding processing is effected with respect tothe thinned-out image.

The basic principle of the present invention, if expressed by a formula,can be written as shown below. It is now assumed that all the frequencycomponents v(f) of the image to be encoded become 0s above a certainfrequency f_(s).

[Mathematical Formula 4]

v(f)=0(f>f _(s))  (5)

The frequency fs can be determined by frequency analysis. Meanwhile, ifit is assumed that the pitch of the image format is p, a maximumfrequency f_(max) which can be expressed becomes as follows, asdescribed above.

[Mathematical Formula 5]

f _(max)=1/T_(max)=1/2P  (6)

Naturally, f_(s)≦f_(max). The pitch p_(s) of the resolution necessaryfor expressing f_(s) can be obtained from the following formula in thesame way as Formula (6).

[Mathematical Formula 6]

p _(s)=T_(s)/2=1/2f_(s)  (7)

At this time, f_(s)≦f_(max); therefore, p≧p_(s). This very pitch p_(s)indicates the maximum resolution referred to in the present invention.

The advantage of the present invention can be qualitatively explained asfollows. In a case where an input is an artificial image, sinceinformation is contained in large volumes in high-frequency componentsas can be seen from FIG. 27, most of the pixels cannot be thinned out.However, since the image can be sufficiently compressed reversibly bypredicative encoding or the like as described in the description of thesecond conventional example, no problem is presented even if such pixelscannot be thinned out. In addition, in a case where the input is anatural image, the high-frequency components may be quantized to someextent, as described in the first conventional example. Accordingly, thehigh-frequency components which are small in some measure may beignored, and since the maximum frequency can be lowered, the necessaryresolution, i.e., the number of pixels, can be made small as a result.

From the perspective of the present invention, the problems of theconventional examples can be expressed as follows. In the firstconventional example, the quantization of frequency components iseffected irrespective of the resolution which an image shouldessentially possess. Since the quantization of frequency components isan act which ignores small components as far as a high-frequency regionis concerned, the quantization of frequency components is equivalent toforcibly lowering-the resolution. Accordingly, this results in eitherthe degradation of image quality with respect to an artificial image,for which a maximum resolution is partially required, or finequantization with a resultant increase in the amount of codes.

On the other hand, the second conventional example is not able to reducethe amount of codes since a natural image is encoded with a highresolution which is not essentially required.

Further, in the third conventional example, since lossy encoding andlossless encoding are effected separately in a frequency space and apixel value space which are utterly different spaces, problems pointedout in Description of the Related Art occur. In this respect, in thepresent invention, since all the image is processed from the commonperspective of the resolution, such a distortion does not occur.

The schematic configuration of the present invention is as follows. Thepresent invention is based on lossless predictive coding, and as for anatural image which requires quantization, loss (irreversibility) isrealized by subsampling pixels in its prior stage. The subsamplingprocessing of pixels is effected while determining whether the givenimage has an optimum resolution by means of the frequency analysis andquantization. With respect to an artificial image, on the other hand,since subsampling processing is ineffective, quantization is carried outstrictly, and only the pixels which are not required for losslessencoding are thinned out.

Next, a specific description will be given of the operating principle ofthis embodiment. In this embodiment, DCT is used as simplified frequencyanalysis. DCT and its inverse transformation are expressed by Formulae(1) and (2) above. That is, the DCT coefficient y(u, v) is the linearsum of pixel values x(m, n), and in the case of 8×8 DCTs one DCTprocessing is expressed by writing out 64 formulae.

Here, the DCT coefficient corresponds to the frequency component withina block. Accordingly, if the fact that a certain block does not have ahigh-frequency component is expressed by a formula, and if it is assumedthat u- and v-direction maximum frequencies are f_(u) and f_(v),respectively, this fact can be expressed as Formula (8) (where (0≦f_(u),f_(v)≦7).

[Mathematical Formula 7]

y(u, v)=0(u>f _(u) or v>f _(v))  (8)

Since the number of DCT coefficients y(u, v) which satisfy Formula (8)is (64−(f_(u)+1)×(f_(v)+1)), the left sides of (64−(f_(u)+1)×(f_(v)+1))formulae become 0. This corresponds to a reduction in the number ofunknowns from 64 to (f_(u)+1)×(f_(v)+1) if DCT processing is consideredas a simultaneous system of linear equations in which arguments arepixel values and unknowns are DCT coefficients. That is, since(64−(f_(u)+1)×(f_(v)+1)) formulae become redundant,(64−(f_(u)+1)×(f_(v)+1)) pixel values among the pixel values which arearguments can be eliminated by the operation of the formulae.Consequently, it can be appreciated that if only (f_(u)+1)×(f_(v)+1)pixel values are known, 64 DCT coefficients, hence, the pixel values,can be reconstructed afterwards by solving the simultaneous system oflinear equations involved in DCT processing.

However, calculation accuracy is not taken into consideration here. Inaddition, although a description has been given of 64 simultaneousequations for simplification's sake, processing may be effected byconsidering the case as a combination of eight simultaneous equationswhich are two-dimensionally independent in the light of the nature ofthe two-dimensional DCT. In addition, the above facts also hold true ofDCTs other than the 8×8 DCTs with the exception of constants.

In accordance with the above-described theory, ny pixels can be thinnedout from the 8×8 blocks, but restrictions are imposed on the method ofsubsampling. Since the two-dimensional DCT is effected by a combinationof one-dimensional DCTs, subsampling must be effected such that(f_(u)+1)×(f_(v)+1) pixels ultimately remain. However, in aconfiguration which is two-dimensionally independent and in whichinterpolation in the u-direction, for instance, is effected first, itsuffices if (f_(v)+1) pixels remain in the v-direction when theinterpolation in the u-direction interpolation is finished. Although norestriction is imposed on the interval of the pixels at this time, sincethe pixel values can be provided with only integer accuracy, ifspatially close pixels are left, there are cases where the accuracy ofvalues of interpolated pixels declines.

The foregoing logic is expressed by a formula by using one-dimensionalDCT of 8 pixels for the sake of simplicity. First, as a transformationformula of one-dimensional DCT, Formula (9) can be readily derived fromFormula (1). $\begin{matrix}\left\lbrack {M\quad a\quad t\quad h\quad e\quad m\quad a\quad t\quad i\quad c\quad a\quad l\quad F\quad o\quad r\quad m\quad u\quad l\quad a\quad 8} \right\rbrack & \quad \\{{y(u)} = {\frac{c(u)}{2}{\sum\limits_{m = 0}^{7}{\left( {{x(m)} - 128} \right)\cos \quad \frac{\left( {{2n} + 1} \right)v\quad \pi}{16}}}}} & (9)\end{matrix}$

Since Formula (9) is in the form of a mere sum of products, Formula (9)can be-expressed as a matrix. If the term of cos is expressed as d(u,m), we obtain Formula (10). $\begin{matrix}\left\lbrack {M\quad a\quad t\quad h\quad e\quad m\quad a\quad t\quad i\quad c\quad a\quad l\quad F\quad o\quad r\quad m\quad u\quad l\quad a\quad 9} \right\rbrack & \quad \\{\begin{pmatrix}{y(0)} \\{y(1)} \\\vdots \\{y(7)}\end{pmatrix} = {\frac{c(u)}{2}\begin{pmatrix}{d\left( {0,0} \right)} & {d\left( {0,1} \right)} & \ldots & {d\left( {0,7} \right)} \\{d\left( {1,0} \right)} & {d\left( {1,1} \right)} & \ldots & \quad \\\vdots & \vdots & \quad & \vdots \\{d\left( {7,0} \right)} & \quad & \ldots & {d\left( {7,7} \right)}\end{pmatrix}\begin{pmatrix}{{x(0)} - 128} \\{{x(1)} - 128} \\\vdots \\{{x(7)} - 128}\end{pmatrix}}} & (10)\end{matrix}$

Here, if it is assumed that f_(u)=2, then y(u)=0 (u>2), so that Formula10 is rewritten as $\begin{matrix}\left\lbrack {M\quad a\quad t\quad h\quad e\quad m\quad a\quad t\quad i\quad c\quad a\quad l\quad F\quad o\quad r\quad m\quad u\quad l\quad a\quad 10} \right\rbrack & \quad \\{\begin{pmatrix}{y(0)} \\{y(1)} \\{y(2)} \\0 \\\vdots\end{pmatrix} = {\frac{c(u)}{2}\begin{pmatrix}{d\left( {0,0} \right)} & {d\left( {0,1} \right)} & \ldots & {d\left( {0,7} \right)} \\{d\left( {1,0} \right)} & {d\left( {1,1} \right)} & \ldots & \quad \\\vdots & \vdots & \quad & \vdots \\{d\left( {7,0} \right)} & \quad & \ldots & {d\left( {7,7} \right)}\end{pmatrix}\begin{pmatrix}{{x(0)} - 128} \\{{x(1)} - 128} \\\vdots \\{{x(7)} - 128}\end{pmatrix}}} & (11)\end{matrix}$

Since the left sides of the lower five equations in Formula (11) arefixed at 0, if they are substituted in the upper three equations, thevariables on the right side can be deleted. For example, if x(7) isdeleted from x(3), Formula (12) can be obtained as a result.$\begin{matrix}\left\lbrack {M\quad a\quad t\quad h\quad e\quad m\quad a\quad t\quad i\quad c\quad a\quad l\quad F\quad o\quad r\quad m\quad u\quad l\quad a\quad 11} \right\rbrack & \quad \\{\begin{pmatrix}{y(0)} \\{y(1)} \\{y(2)}\end{pmatrix} = {\frac{c(u)}{2}\begin{pmatrix}{d^{\prime}\left( {0,0} \right)} & {d^{\prime}\left( {0,1} \right)} & {d^{\prime}\left( {0,2} \right)} \\{d^{\prime}\left( {1,0} \right)} & {d^{\prime}\left( {1,1} \right)} & {{d^{\prime}\left( {1,2} \right)}\quad} \\{d^{\prime}\left( {2,0} \right)} & {{d^{\prime}\left( {2,1} \right)}\quad} & {d^{\prime}\left( {2,2} \right)}\end{pmatrix}\begin{pmatrix}{{x(0)} - 128} \\{{x(1)} - 128} \\\vdots \\{{x(2)} - 128}\end{pmatrix}}} & (12)\end{matrix}$

If the three pixel values of x(0), x(1), and x(2) can be known fromFormula (12), y(0), y(1), and y(2) can be determined. Since it is knownthat y(3) through y(7) are 0, x(3) through x(7) can then be interpolatedby an inverse transformation of Formula (9). Since no restrictions areimposed on the manner of selection of variables that are deleted inFormula (11), any combination may be adopted as the pixel values whichare selected for the right side of Formula (12) insofar as the number ofpixel values agrees. Nevertheless, there is the nature that the widerthe interval as described above, the higher the accuracy of theinterpolation.

An example of the method subsampling is shown in FIGS. 5A to 5D. FIG. 5Aclearly satisfies the above-described conditions. In FIG. 5B, decodingis possible by first effecting interpolation in the u-direction and theneffecting interpolation in the v-direction. Neither FIG. 5C nor FIG. 5Dsatisfies the restrictions.

A description will be given of the extension of the subsampling method.In the above, for the sake of simplicity a description has been given ofsubsampling which is based on Formula (8). In fact, Formula (8)expresses well the concept of the present invention which has beendescribed in the beginning of this embodiment. In this embodiment,however, since the interpolation processing can be reduced to asimultaneous system of equations, Formula (8) can be extended. That is,even if the frequency is f_(u) or less, in a case where a frequencyfs_(u) whose component becomes 0 irrespective of the v component, eightequations concerning the frequency fs_(u) can be deleted from thesimultaneous system of equations. Therefore, the number of pixels to beleft in the u direction can be reduced to f_(u). fs_(u) may be plural.Also, the same holds true of the v direction.

In addition, although subsampling processing which is independentlybased on f_(u) and f_(v) was effected in the above, both axes may beadjusted to the higher one of the frequencies f_(u) and f_(v). As aresult, the number of pixels which can be thinned out decreases, butsince the patterns which are thinned out also decrease, processing suchas coefficient analysis processing and interpolation processing can besimplified. Of course, if the degradation of the image quality isallowed, values such as average values or minimum values of f_(u) andf_(v) may be used. Still alternatively, a similar advantage can beobtained if appropriate quantization is effected by using any of 0, 1,3, and 7 as the values of f_(u) and f_(v).

Incidentally, since Formula (8) is written as being dependent upon themaximum frequencies in the f_(u)- and f_(v)-directions, an area ofeffective frequency components forms a rectangle on the DCT coefficientblocks. This is ascribable to the fact that the two-dimensional DCT isrealized by a combination of one-dimensional DCTs. If the bases of thetwo-dimensional transforms are completely independent from each other,the area can be extended to a free shape other than the- rectangle. Forinstance, a restriction may be imposed in such a manner as to leave onlyupper left triangular components in the frequency components. In thiscase, there are no longer restrictions on the subsampling method.

A description will be given of the quantization of the DCT coefficients.As described in the description of the JPEG-DCT method, in the frequencytransform coding, the amount of codes can be reduced in a state in whichthe degradation of the image quality is suppressed by coarselyquantizing high-frequency components. In this embodiment as well, it ispossible to apply the quantization processing using, for example, thequantization table shown in FIGS. 33A and 33B. Although the frequencycomponents which become 0 as a result of quantization increase, theabove-described basic principle can be applied substantially as it is.

Hence, a description will be given of a specific procedure ofapplication of quantization. In the present invention, the frequencytransform is used only in the analysis of an image, and actualquantization is realized by subsampling of pixels. Accordingly, thequantization of frequency components is, to be strict, realized bythreshold processing with respect to absolute values. That is,processing is effected in which each frequency component is comparedwith a quantization step, and if smaller, the frequency component is setto 0. If the quantization table is set appropriately, by subjectingthreshold-processed coefficient data to inverse DCT processing, it ispossible to obtain an image which is free of the degradation of imagequality and in which high-frequency components are limited. Thereafter,it suffices if the above-described basic principle is applied as it is.To sum up, the following procedure is taken.

(Algorithm When Quantizing Coefficient Data)

Step 1: DCT processing is effected.

Step 2: The coefficient data is subjected to threshold processing, andcomponents which are smaller than the quantization step is set to 0.Maximum frequency components at this time are set as f_(u) and f_(v).

Step 3: Inverse DCT processing is effected.

Step 4: Subsampling processing is effected on the basis of f_(u) andf_(v) determined in Step 2. If the image remains, the operation returnsto Step 1.

In Step 2, components which happen to be subjected to thresholdprocessing to 0 are generated at frequencies below f_(u) and f_(v). Inthis algorithm, since the subsampling processing in Step 4 is based-onf_(u) and f_(v) determined in Step 2, even if such components arecompulsively not set to 0, the amount of processing does not change inboth subsampling and interpolation processing. Accordingly, thefollowing processing may be interposed between Step 2 and Step 3.

Among frequency components below f_(u) and f_(v), if there are thosewhich have been subjected to threshold processing, such frequencycomponents are returned to data persisting prior to thresholdprocessing.

In the present invention, a compressing means for predictive coding orthe like is assumed in a later stage. Since the number of pixels whichare sent to the later stage can be reduced on the basis of theabove-described basic principle, the processing in the later stage canbe alleviated as an auxiliary effect of the present invention. Sincethis is effective when image processing, such as color transform,enlargement/reduction, rotation, and clipping, is effected in a laterstage, the present invention can be applied as an accelerator for imageprocessing.

Since a description has been given of the basic principle, a specificdescription will now be given of this embodiment. Hereafter, adescription will be given of a portion for effecting the subsampling ofpixels with respect to a natural image, excluding the aforementionedlater stage.

FIGS. 1 and 2 are block diagrams illustrating a first embodiment of thepresent invention. In the drawings, portions which are similar to thoseof FIGS. 29 and 30 will be denoted by the same reference numerals, and adescription thereof will be omitted. In FIGS. 1 and 2, reference numeral30 denotes a coefficient analyzing unit; 40, a coefficient analysisoutput unit; 50, a high-frequency coefficient masking unit; 60, aninverse DCT unit; 70, a pixel subsampling unit; 80, a subsampled-imageoutput unit; 130, analyzed coefficient data; 140, low-frequencycoefficient data; 150, low-frequency image data; 160, subsampled imagedata; 210, a subsampled-image input unit; 220, a coefficient analysisinput unit; 230, a coefficient interpolating unit; and 310, interpolatedcoefficient data.

A description will be given of the various units shown in FIGS. 1 and 2.The encoding apparatus shown in FIG. 1 has the following configuration.The coefficient analyzing unit 30 makes a comparison betweenpredetermined constants and coefficient data 120, and sends the resultsof comparison as the analyzed coefficient data 130 to the coefficientanalysis output unit 40, the high-frequency coefficient masking unit 50,and the pixel subsampling unit 70, respectively. The coefficientanalysis output unit 40 outputs the analyzed coefficient data 130 to anexternal circuit. The high-frequency coefficient masking unit 50replaces some of the high-frequency coefficients of the coefficient data120 with 0s on the basis of the analyzed coefficient data 130, and sendsthe same as the low-frequency coefficient data 140 to the inverse DCTunit 60. The inverse DCT unit 60 effects inverse DCT processing, whichis the inverse transformation of DCT processing effected by the DCT unit20, with respect to the low-frequency coefficient data 140, and sendsthe result to the pixel subsampling unit 70 as the low-frequency imagedata 150. The pixel subsampling unit 70 effects subsampling processingwith respect to the low-frequency image data 150 on the basis of apreset subsampling method and the analyzed coefficient data 130, andsends the result to the subsampled-image output unit 80 as thesubsampled image data 160. The subsampled-image output unit 80 sends thesubsampled image data 160 to an external circuit.

Next, the decoding apparatus shown in FIG. 2 has the followingconfiguration. The subsampled-image input unit 210 receives thesubsampled image from the external circuit, and sends the same to thecoefficient interpolating unit 230 as the subsampled image data 160. Thecoefficient analysis input unit 220 receives the analyzed coefficientdata from the external circuit, and sends the same to the coefficientinterpolating unit 230 as the analyzed coefficient data 130. Thecoefficient interpolating unit 230 effects interpolation processing ofthe DCT coefficients with respect to the subsampled image data 160 onthe basis of the analyzed coefficient data 130, and sends the result toan inverse DCT unit 240 as the interpolated coefficient data 310. Theinverse DCT unit 240 effects inverse DCT processing with respect to theinterpolated coefficient data 310, and sends the result to adecoded-image output unit 250 as decoded image data 320.

A description will be given of the operation of the first embodiment onthe basis of the above-described configuration. FIGS. 3 and 4 areflowcharts illustrating the operation of the first embodiment of thepresent invention.

First, a description will be given of the encoding procedure of thisembodiment with reference to FIG. 3. In S10, an image is inputted to animage input unit 10 from an external circuit, thereby obtaining inputimage data 110. In S20, DCT processing is effected in a DCT unit 20,thereby obtaining the coefficient data 120. In S30, the coefficientanalyzing unit 30 makes a comparison between the coefficient data 120and the predetermined constants, and obtains the result as the analyzedcoefficient data 130. In S40, the high-frequency coefficient maskingunit 50 replaces some of the high-frequency coefficients of thecoefficient data 120 with 0s on the basis of the analyzed coefficientdata 130, and sets the same as the low-frequency coefficient data 140.In S50, the inverse DCT unit 60 effects inverse DCT processing withrespect to the low-frequency coefficient data 140, thereby obtaining thelow-frequency image data 150. In S60, the pixel subsampling unit 70effects the subsampling processing of pixels on the basis of theanalyzed coefficient data 130, thereby obtaining the subsampled imagedata 160. In S70, the coefficient analysis output unit 40 and thesubsampled-image output unit 80 respectively output the analyzedcoefficient data 130 and the subsampled image data 160 to externalcircuits. In S80, a determination is made as to whether or not all theinput image data 110 inputted in S10 has been processed, and ifunprocessed data remains, the operation returns to S10, while if all theinput image data 110 has been processed, the encoding procedure ends.

Next, a description will be given of the decoding procedure of thisembodiment with reference to FIG. 4. In S110, the subsampled-image inputunit 210 and the coefficient analysis input unit 220 respectivelyreceive the subsampled image data 160 and the analyzed coefficient data130 from the external circuits. In S120, the coefficient interpolatingunit 230 obtains the interpolated coefficient data 310 on the basis ofthe subsampled image data 160 and the analyzed coefficient data 130. InS130, the inverse DCT unit 240 effects inverse DCT processing withrespect to the interpolated coefficient data 310, thereby obtaining thedecoded image data 320. In S140, the decoded-image output unit 250outputs the decoded image data 320 to an external circuit. In S150, adetermination is made as to whether or not all the subsampled image data160 and analyzed coefficient data 130 which were inputted in S110 havebeen processed, and if unprocessed data remains, the operation returnsto S110, while if all the inputted data have been processed, thedecoding procedure ends.

A description will be given of the coefficient analyzing processing inthe above-described operation. In the coefficient analyzing processing,constants which are coarse with respect to high-frequency coefficients,as in quantization tables used in the JPEG-DCT method, are used.However, the effectiveness of the DCT coefficients is judged not by thequantization as described above but by mere threshold processing.

Referring to FIGS. 6A to 6E, a description will be given of the flow ofcoefficient analysis processing in a case where a recommended table ofthe JPEG-DCT method is used. FIG. 6A is an example of the coefficientdata 120 obtained by DCT processing. If the quantization of the JPEG-DCTmethod is effected with respect to this coefficient data, quantizedcoefficient data shown in FIG. 6B is obtained. Since thresholdprocessing is effected in this embodiment, if effective coefficients areexpressed by 1s and ineffective coefficients by 0s, the analyzedcoefficient data 130 such as the one shown in FIG. 6C is obtained.

As described in the beginning of the description of this embodiment,since the number of pixels which can be thinned out is determined bymaximum frequencies in the u- and v-directions, even if this informationis set as shown in FIG. 6D, necessary information is not lost. Inaddition, as a format, the data may be abbreviated such as (4, 4). Thelow-frequency coefficient data 140 which is prepared by thehigh-frequency coefficient masking unit 50 on the basis of this analyzedcoefficient data 130 becomes as shown in FIG. 6E.

In addition, during the operation, the coefficient interpolationprocessing which is effected by the coefficient interpolating unit 230is carried out by solving the simultaneous system of equations, asdescribed in the beginning of the description of this embodiment.Incidentally, as for the simultaneous system of equations which areselected, there is only a combination of 64 simultaneous equations inthe case of, for example, 8×8 blocks, so that if an inverse matrix isdetermined in advance, processing can be effected simply.

In addition, during the operation, as it has already been described thatit is assumed that the subsampled image data 160, which is outputted inS70, is encoded by the encoding apparatus in a later stage, the analyzedcoefficient data 130 may be encoded by some entropy coding in a similarmanner.

As described above, in accordance with this embodiment, it is possibleto effectively encode a natural image irreversibly by using losslessencoding in a later stage. In encoding an artificial image, it sufficesif all the values of the quantization table used in thresholdprocessing, which is effected by the coefficient analyzing unit 30 ofthe encoding apparatus shown in FIG. 1, are set to 0s. Since thequantization of the coefficient data is not carried out as a result, ifthere are no calculation errors, the encoding apparatus shown in FIG. 1operates as a lossless encoding apparatus. Of course, it is possible toseparately provide a data path for bypassing the configuration shown inFIG. 1 when an artificial image is inputted.

In addition, the processing in the high-frequency coefficient maskingunit 50 and the inverse DCT unit 60 of the encoding apparatus shown inFIG. 1, in final analysis, becomes the same processing as cutting ahigh-frequency region of the image inputted by the image input unit 10,i.e., low-pass filter processing. Accordingly, it is also possible tomake the high-frequency coefficient masking unit 50 and the inverse DCTunit 60 low-pass filters.

(First Example of Extension)

A description will be given of the extension of this embodiment. As apoint of difference in the decoded image between the JPEG-DCT method andthis embodiment, it is possible to cite that in contrast to the factthat in the JPEG-DCT method all the pixel values can possibly varysubtly from those of the input image, in this embodiment the pixelswhich were not thinned out are sent as they are to the decoding side. Asan example, average absolute values of differences between a decodedimage and an input image in accordance with the JPEG-DCT method weretaken with respect to a number of natural images. FIG. 7 shows theresults in which these average absolute values were classified in layersaccording to the results of analysis of the coefficient data. Theresults of coefficient analysis on the abscissa were computed by thefollowing formula.

[Mathematical Formula 12]

(Analysis type)=f _(u) +f _(vx)8  (13)

In the above-described example, since the image subjected to inverse DCTprocessing is outputted as it is as a decoded image, there are caseswhere even in the case of pixels which were not thinned out pixel valuesdeviate due to calculation errors and the like. In this embodiment,before this decoded image is outputted, the pixels which were notthinned out can be newly returned to their intrinsic pixel values. Inthe case where such processing is interposed, an utterly identicalsubsampled image is obtained by effecting subsampling in a similarmanner when re-coding is effected. Accordingly, even if encoding anddecoding are repeated, it is possible to realize lossy encoding in whichthe degradation of the image quality does not overlap. This is the firstexample of extension of this embodiment.

To thin out the same pixels as those of the initial encoding at the timeof re-coding, it suffices if information on the subsampling method isincluded in the coefficient information data 130 and the subsampledimage data 160. FIG. 8 is an example of such a data format. In thisexample, data concerning a subsampling technique is included as aheader. This subsampling technique data may be one in which subsamplingmethods corresponding to the coefficient information data 130 areenumerated as shown in FIG. 9, or may be one in which preset IDs aresimply indicated. It goes without saying that if encoding and decodingare repeated by an encoding apparatus in which subsampling methods arenot dependent upon conditions other than the coefficient informationdata 130, such a scheme is not necessary.

FIG. 10 shows a schematic diagram of a decoding apparatus in the firstexample of extension of this embodiment. In the drawing, those portionswhich are similar to those of FIG. 1 are denoted by the same referencenumerals, and a description thereof will be omitted. Reference numeral241 denotes a pixel-value correcting unit, and numeral 321 denotescorrected decoded image data.

A description will be given of the respective units shown in FIG. 10. Ofthe decoded image data 320, the pixels imparted by the subsampled imagedata 160 are replaced with the pixel values of the subsampled image data160 by the pixel-value correcting unit 241. The result is sent to thedecoded-image output unit 250 as the corrected decoded image data 321.Since the description of other portions and of the operation is largelysimilar to the above-described description, a description thereof willbe omitted.

(Second Example of Extension)

Returning to the extension of this embodiment, if consideration is givenby following the pattern of the JPEG-DCT method as seen in FIG. 7referred to earlier, it can be expected that even if pixel values arequantized during predictive encoding in a stage following thisembodiment, not much influence will be exerted on the image quality.Since it can be considered that the quantization which is allowed hereis dependent upon the threshold processing which is effected withrespect to frequency components, if the two processes are controlled inassociation with each other, efficient quantization is possible. Theexample of FIG. 7 referred to earlier is the result in which thresholdprocessing is effected fixedly by using the quantization table shown inFIG. 33A, and it is possible to ascertain the tendency of the absolutevalues of decoding errors through the results of analysis of thecoefficient data. The pixel-value quantization processing may beeffected by using this as a reference. For example, if it is estimatedthat the quantization step is two times the average of error values onthe assumption that errors occur with a uniform distribution, Formula(14) holds for each analysis type.

[Mathematical Formula 13]

(Quantization step)=(average of absolute values of decodingerror)×2  (14)

If the error distribution has a deviation centering on 0, for example, 2in Formula (14) may be a slightly smaller value. At any rate, this valuecan be experimentally calculated by statistical processing. Of course,Formula (14) may be calculated by nonlinear operation by incorporating amore complicated assumption. Described above is the second example ofextension of this embodiment.

FIG. 11 shows a schematic diagram of an encoding apparatus in the secondexample of extension of this embodiment. In the drawing, those portionswhich are similar to those of FIG. 1 are denoted by the same referencenumerals, and a description thereof will be omitted. Reference numeral71 denotes a pixel-value quantizing unit, and numeral 161 denotesquantized subsampled image data.

A description will be given of the respective units shown in FIG. 11.The pixel-value quantizing unit 71 quantizes pixel values with respectto the decoded image data 160 by a predetermined technique on the basisof the threshold values used in the coefficient analyzing unit 30 andthe analyzed coefficient data 130, and sends the result to thesubsampled-image output unit 80 as the quantized decoded image data 161.Since the description of other portions and of the operation is largelysimilar to the above-described description, a description thereof willbe omitted.

In the above-described example, the quantization in the pixel-valuequantizing unit 71 may, of course, be fixed without being related tothreshold values and the analyzed coefficient data 130.

(Third Example of Extension)

A description will be given of a third example of extension of thisembodiment. It has already been described that both lossy encoding andlossless encoding can be executed by controlling the threshold valuesused in coefficient analysis processing. Hereafter, a modification inwhich such threshold control is effected dynamically will be describedas the third example of extension.

FIG. 12 is a schematic diagram of the third example of extension of thisembodiment. In the drawing, reference numeral 31 denotes an imagedetermining unit, and numeral 131 denotes threshold control data.

A description will be given of the respective units shown in FIG. 12.The image determining unit 31 determines an image with respect to theinput image data 110 by a predetermined technique, and sends the resultto the coefficient analyzing unit 30 as the threshold control data 131.Since the description of other portions and of the operation is largelysimilar to the above-described description, a description thereof willbe omitted.

The image determining unit 31 determines a natural image and anartificial image. Specifically, the presence or absence of noise can beestimated from the manner of spread of the distribution of pixel values,entropy of lower bits, the sharpness of the edge, and the like, so thatthe determination is made on the basis thereof. Although notillustrated, a configuration may be provided such that a distinctionbetween a natural image and an artificial image is sent from an externalcircuit as side information. In this case, the image determining unit 31effects transform of the side information into the threshold controldata 131.

Such threshold control may be effected for each image or for eachlocation of the image. In addition, control may be provided on the basisof local nature of the image without adhering to the distinction betweenthe natural image and the artificial image. For instance, thedistribution of pixel values and frequency components, the sharpness ofthe edge, the presence or absence of a pattern, the presence or absenceof a fine line, the presence or absence of a gradation, and the like canserve as indices.

(First Simplification)

Next, a description will be given of the simplification of thisembodiment. In the decoding apparatus shown in FIG. 2, a means fordirectly interpolating the pixel values may be provided instead of thecoefficient interpolating unit 230 and the inverse DCT unit 240. Themeans for interpolating the pixel values referred to herein may be anyinterpolating means insofar as it is capable of interpolating the pixelvalues, such as the nearest-neighbor interpolation, 4-point linearinterpolation, and 9-point second-order interpolation. In this case,since the basic principle of this embodiment still does not apply, thedegradation of the image quality is unavoidable. In principle, however,the coefficient interpolation processing which is carried out in thisembodiment has an effect similar to that of a low-pass filter on the DCTcoefficients, and the pixel-value interpolation processing enumeratedabove also has an effect similar to that of a low-pass filter, so thatthe two approximations can be regarded as very simple approximations.Described above is the first example of simplification of thisembodiment.

FIG. 13 is a schematic diagram of the first example of simplification ofthis embodiment. In the drawing, those portions which are similar tothose of FIG. 2 are denoted by the same reference numerals, and adescription thereof will be omitted. Reference numeral 231 denotes apixel-value interpolating unit.

A description will be given of the respective units shown in FIG. 13.The pixel-value interpolating unit 231 interpolates the pixels whichwere thinned out with respect to the subsampled image data 160 by apredetermined technique, and sends the result to the decoded-imageoutput unit 250 as the decoded image data 320. Since the description ofother portions and of the operation is largely similar to theabove-described description, a description thereof will be omitted.

As described above, since the pixel-value interpolation in the firstexample of simplification has an effect similar to that of a low-passfilter, even if the same number of pixels are thinned out, thispixel-value interpolation is dependent upon the pixels which areselected, so that there is a possibility of causing a difference in theimage quality. In addition, it has already been described that ifrestrictions are observed, the selection of pixels can be effectedfreely to some extent. Accordingly, if, for example, adjustment is madein such a manner as to select peak values within the blocks whenselecting the pixels to be left, it becomes possible not to impairdynamic ranges of the blocks.

In addition, when the pixel-value interpolation is effected, there is noneed for Formula (8) to hold, the image subject to subsamplingprocessing may not have its high-frequency region restricted with theexception of the problem of aliasing distortion. Accordingly, in theencoding apparatus shown in FIG. 1, for example, the input image data110 may be sent directly to the pixel subsampling unit 70 instead ofproviding the high-frequency coefficient masking unit 50 and the inverseDCT unit 60. In this case, processing can be simplified substantially.This schematic diagram is shown in FIG. 14. The description of variousparts and the operation will be omitted. Incidentally, in a case whereno quantization is effected with respect to the high-frequencycomponents by the coefficient analyzing unit 30 shown in FIG. 1, theinput image already satisfies Formula (8). Therefore, the encodingapparatus can still be realized by the configuration shown in FIG. 14without affecting the image quality and the amount of codes.

Further, also in a case where the pixel-value interpolation is effectedby the decoding apparatus, a decoded image can be simulated by theencoding apparatus. Accordingly, instead of the coefficient analyzingunit 30 shown in FIG. 1, it is possible to provide a means whichsimulates the decoded image data 320 with pixel values interpolated, anddetermines the coefficient information data 130 while evaluating errorswith the input image data 110. The evaluation of errors may be effectedon the basis of the signal-to-noise (SN) ratio, a maximum value oferrors, the variance, the dynamic range, or the like. In this case, thecoefficient information data 130 simply means the pixel sampling ratio.Since the frequency analysis is not effected in this case, the DCT unit20 can be clearly omitted. Since the configuration can be analogized, aschematic diagram and the rest of the description will be omitted.

(Second Simplification)

Next, a case is considered in which, instead of the image, the DCTcoefficients are received as an input. For example, in a case where animage encoded by the JPEG-DCT method is received, data which areobtained by corresponding entropy decoding are not pixel values but DCTcoefficients. In such a case, it suffices if the DCT coefficients areinputted directly to the coefficient analyzing unit 30 and thehigh-frequency coefficient masking unit 50 of the encoding apparatusshown in FIG. 1. In this way, the DCT unit 20 and its processing can beomitted. This is a second example of simplification.

FIG. 15 is a schematic diagram of the second example of simplificationof this embodiment. In the drawing, those portions which are similar tothose of FIG. 1 are denoted by the same reference numerals, and adescription thereof will be omitted. Reference numeral 11 denotes a codeinput unit; numeral 21 denotes an entropy decoding unit; and numeral 111denotes encoded data.

A description will be given of the respective units shown in FIG. 15.The code input unit 11 receives a code from an external circuit as aninput, and sends the same to the entropy decoding unit 21 as the encodeddata 111. The entropy decoding unit 21 decodes the encoded data 111, andsends the same to the coefficient analyzing unit 30 and thehigh-frequency coefficient masking unit 50 as the coefficient data 120.Since the description of other portions and of the operation is largelysimilar to the above-described description, a description thereof willbe omitted.

In the case where the DCT coefficients are thus inputted, a case canalso be assumed in which the DCT coefficients have already beenquantized. In this case, it is necessary to effect inverse quantizationprocessing by the entropy decoding unit 21.

Further, if the quantization step for the inputted code is coarser thanthe quantization step set in advance for the coefficient analyzing unit30, the processing of various units can be simplified. One processingwhich can be simplified concerns the coefficient analysis processing inthe coefficient analyzing unit 30, and since the coefficients which arenot 0s do not become 0s as a result of threshold processing, it sufficesif, instead of performing threshold processing, a determination ismerely made as to whether the frequency components are 0s or other than0s. In addition, for a similar reason, the processing which is effectedby the high-frequency coefficient masking unit 50 and the inverse DCTunit 60 can be omitted without the degradation of the image. Thedecoding apparatus shown in FIG. 15 may be provided with a configurationwhich makes it possible to bypass the aforementioned portions in such acase.

(Third Example of Simplification)

Next, a description will be given of a third example of simplification.According to the configurations shown in FIGS. 1 and 2, inputs andoutputs are made independently with respect to the analyzed coefficientdata 130 and the subsampled image data 160, but the two items of datamay be combined and inputted or outputted. This is the third example ofsimplification.

FIG. 16 is a schematic diagram of the third example of simplification ofthis embodiment. In the drawing, those portions which are similar tothose of FIG. 1 are denoted by the same reference numerals, and adescription thereof will be omitted. Reference numeral 72 denotes a datacomposing unit; 81, a composite-data output unit; and 162, compositedata.

A description will be given of the various units shown in FIG. 16. Thedata composing unit 72 combines the analyzed coefficient data 130 andthe subsampled image data 160, and sends the result to thecomposite-data output unit 81 as the composite data 162. Thecomposite-data output unit 81 outputs the composite data 162 to anexternal circuit. Since the description of other portions and of theoperation is largely similar to the above-described description, adescription thereof will be omitted. In addition, since a decodingapparatus corresponding to the encoding apparatus in accordance with thethird example of simplification can be easily analogized, a descriptionthereof will be omitted.

A description will be given of data composition processing which iseffected by the data composing unit 72. The composite data 162 needs tobe composed in such a manner that it can be decomposed into the analyzedcoefficient data 130 and the subsampled image data 160 by the decodingapparatus. As such an example, a number of examples are conceivable,including a method in which the two items of data are simply combined asshown in FIG. 17 and a method in which the two items of data arecombined in units of blocks as shown in FIG. 18. It goes without sayingthat the two items of data may be combined in other units.

(Fourth Simplification)

Next, a description will be given of a fourth example of simplification.In a case where effective frequency components of an input image areknown in advance, the coefficient information may be designated from anexternal circuit. This is a fourth example of simplification.

FIG. 19 is a schematic diagram of the fourth example of simplificationof this embodiment. In the drawing, those portions which are similar tothose of FIG. 1 are denoted by the same reference numerals, and adescription of the various units and operation will be omitted.

If such a configuration is adopted, in a case where it is clearly knownthat high-frequency components are noise, those components which aredirectly set to 0s can be designated, so that the amount of codesdecreases. As an example of such an image, it is possible to cite anatural image which has been enlarged after being inputted by, forinstance, a low-resolution scanner. Such an enlarged image is sometimessubjected to processing such as edge enhancement so as to suppress theblurring due to enlargement. It cannot be said that high-frequencycomponents which are generated by such processing are noise. However,since such high-frequency components can be reproduced after decoding,it is unnecessary to reproduce such high-frequency components with themaximum resolution referred to in the present invention. In the firstconventional example, on the other hand, even in the case of such animage, reproduction is effected up to a high-frequency region, theamount of codes cannot be reduced. This phenomenon becomes noticeable asthe resolution of an output device improves. This state is shown in FIG.12.

Finally, the results of an experiment in which natural images andartificial images were encoded in accordance with this embodiment areshown in FIG. 20. Further, decoded images of a natural image, which wereprepared in accordance with this embodiment and the first conventionalexample, as well as differential images between an input image and therespective decoded images, are shown in FIGS. 24A to 24C and 25A, 25B,respectively.

(Second Embodiment)

The fundamental concept of the present invention lies in realizing lossyencoding processing in a pixel space by thinning out pixels which can beregarded as having been, so to speak, oversampled in a natural image.This concept can also be extended to a frequency transform techniqueother than DCT. Hereafter, a description will be given of an embodimentin which a general frequency transform technique is used as a secondembodiment of the present invention.

FIGS. 21 and 22 are, respectively, schematic diagrams of an imageencoding apparatus and an image decoding apparatus in accordance withthe second embodiment of the present invention. In the drawings, thoseportions which are similar to those of FIG. 1 are denoted by the samereference numerals, and a description thereof will be omitted. Referencenumeral 22 denotes a frequency transform unit; 61, an inverse transformunit; and 242, an inverse transform unit.

A description will be given of the various units shown in FIGS. 21 and22. The frequency transform unit 22 effects the frequency transform ofthe input image data 110 by some method, and sends the resultantfrequency components to the coefficient analyzing unit 30 and thehigh-frequency coefficient masking unit 50 as the coefficient data 120.The inverse transform unit 61 and the inverse transform unit 242 effectsinverse transform of the frequency transform effected by the frequencytransform unit 22, with respect to the coefficient data 140 and 310,respectively, and output the results as the low-frequency image data 150and the decoded image data 320, respectively. Since the description ofother portions and of the operation is largely similar to thedescription of the first embodiment of the present invention, adescription thereof will be omitted.

In the above-described description, the frequency transform processingmay be any means insofar as it is capable of effecting frequencytransform. For example, the fast Fourier transform, the discrete sinetransform, the subband division, or the like may be employed.

The interpolation processing by the coefficient interpolating unit 230is dependent upon the frequency transform processing which is employed.In the first embodiment, it has been described that interpolation can berealized by solving a simultaneous system of linear equations withrespect to the DCT. A similar technique is applicable to a case where animage is divided into blocks by the fast Fourier transform and to thediscrete sine transform.

In the case of subband division, for example, effectivity is determinedfor each band, and if the components of the bands which are regarded asineffective are set to 0s, high-frequency coefficients can be masked.During interpolation, components of low-frequency bands are restructuredfrom the subsampled image, and if the high-frequency bands are correctedby 0s, coefficient interpolation can be realized.

In addition, although it can be said about all frequency transformtechniques, if the interpolation which is effected by the coefficientinterpolating unit 230 is replaced by pixel-value interpolation as inthe configuration of FIG. 13, interpolation processing can be realized,though in a simplified form.

As is apparent from the above description, in accordance with thepresent invention, efficient encoding and decoding processing can berealized by a single apparatus irrespective of the distinction betweennatural images and artificial images. Accordingly, as compared with thecase where two encoding methods are combined, among other advantages,there is an advantage in that page memory can be eliminated. Further,high-speed image processing can be realized by effecting imageprocessing in a later stage of the present invention. Furthermore, it ispossible to realize encoding and decoding processing in which the imagequality does not undergo degradation even by the repetition of encodingand decoding processing. Still further, generally, many actualhigh-resolution input images are enlarged ones of low-resolution images.In such a case, encoding can be effected with an amount of codingsimilar to that for an effective resolution persisting prior toenlargement.

The foregoing description of a preferred embodiment of the invention hasbeen presented for purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed, and modifications and variations are possible in light of theabove teachings or may be acquired from practice of the invention. Theembodiment was chosen and described in order to explain the principlesof the invention and its practical application to enable one skilled inthe art to utilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the claimsappended hereto, and their equivalents.

What is claimed is:
 1. An image decoding apparatus comprising:coefficient-information input means for inputting coefficientinformation; subsampled-image input means for inputting a subsampledimage; coefficient interpolating means for interpolating a frequencycomponent by a predetermined technique involving solving a set of linearequations in correspondence with the subsampled image inputted by saidsubsampled-image input means and the coefficient information inputted bysaid coefficient-information input means; inversely transforming meansfor effecting inverse frequency transform so as to convert the frequencycomponent interpolated by said coefficient interpolating means into animage; and decoded-image output means for outputting the image convertedby said inversely transforming means.
 2. An image decoding apparatusaccording to claim 1, further comprising: image decoding means fordecoding into an image a code subjected to image encoding with respectto the subsampled image, wherein said subsampled-image input meansinputs as the subsampled image the image decoded by said image decodingmeans.
 3. An image decoding apparatus according to claim 2, wherein thedecoding effected by said image decoding means is inverse processing oflossless coding or inverse processing of predictive coding.
 4. An imagedecoding apparatus according to claim 1, further comprising: pixel-valuecorrecting means for replacing a pixel, which is included in thesubsampled image inputted by said subsampled-image input means of theimage converted by said inversely transforming means, with the pixelvalue of the subsampled image, wherein said decoded-image output meansoutputs the image corrected by said pixel-value correcting means.
 5. Animage decoding apparatus according to claim 1, wherein the frequencytransform effected by said inversely transforming means and saidinversely transforming means is one of discrete cosine transform,Fourier transform, discrete sine transform, subband transform, andwavelet transform.
 6. An image decoding apparatus according to claim 1,wherein the coefficient interpolation effected by said coefficientinterpolating means is one of the solving of a simultaneous system oflinear equations concerning frequency coefficients and pixel values,computation of an inverse matrix determined in advance with respect tothe simultaneous system of linear equations concerning frequencycoefficients and pixel values, and low-pass filtering of the subsampledimage or approximate processing.
 7. An image decoding method comprising:inputting coefficient information; inputting a subsampled image;interpolating a frequency component by a predetermined techniqueinvolving solving a set of linear equations in correspondence with theinputted subsampled image and the inputted coefficient information;effecting inverse frequency transform so as to convert the interpolatedfrequency component; and outputting a converted image.